Papers with positional encoding
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2025) show a tendency to allocate disproportionate attention mass to early (often first) positions independent of semantic content. |
| Approach: | They find that Transformers display an attention sink: disproportionate attention to the first position. |
| Outcome: | The proposed sinks are found in GPT-2–style models with learned query biases and absolute positional embeddings. |
Transformer Language Models without Positional Encodings Still Learn Positional Information (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using positional embeddings, Causal transformer language models learn an implicit notion of absolute positions. |
| Approach: | They propose to use positional embeddings to encode positional information in transformer language models. |
| Outcome: | The proposed model learns an implicit notion of absolute positions across datasets, model sizes, and sequence lengths. |
Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models using Set Encoding (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models typically track the order of tokens using positional encoding, which causes two significant limitations: 1. Positional Bias: When processing long text sequences, the number of token can exceed the range the model was trained on. |
| Approach: | They propose a method that allows multiple pieces of text to be encoded in the same position, eliminating positional bias entirely. |
| Outcome: | The proposed method eliminates positional bias entirely and increases the size of the input an LLM can handle. |
Modeling with Recurrent Neural Networks for Open Vocabulary Slots (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to filling slots that take on values from a virtually unlimited set have been lacking in the natural language area. |
| Approach: | They propose a new attention-based recurrent neural network (RNN) model that captures the concept: Understanding the role of a word may vary according to how long a reader focuses on a particular part of . sentence. |
| Outcome: | The proposed model outperforms existing models with respect to discovering ‘open-vocabulary’ slots without any external information, such as a named entity database or knowledge base. |
Enhancing Transformers for Generalizable First-Order Logical Entailment (2025.acl-long)
Copied to clipboard
Tianshi Zheng, Jiazheng Wang, Zihao Wang, Jiaxin Bai, Hang Yin, Zheye Deng, Yangqiu Song, Jianxin Li
| Challenge: | Moreover, transformers have demonstrated proficiency in logical reasoning over natural language. |
| Approach: | They propose a logic-aware architecture that improves the performance in generalizable first-order logical entailment by combining distribution shifts and unseen knowledge. |
| Outcome: | The proposed architecture outperforms methods designed specifically for knowledge graph query answering on a dataset with a large dataset. |
Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for simultaneous machine translation (SiMT) are more challenging since the source sentence is always incomplete during translating. |
| Approach: | They propose a framework to reduce the position bias by bridging the structural gap between SiMT and full-sentence MT. |
| Outcome: | The proposed framework reduces the position bias by bridging the structural gap between SiMT and full-sentence MT. |
Interpreting Positional Information in Perspective of Word Order (2023.acl-long)
Copied to clipboard
| Challenge: | Attention mechanism is a powerful and effective method utilized in natural language processing, but it is insensitive to positional information. |
| Approach: | They propose a weight concatenation operation to evaluate its efficacy in machine translation tasks. |
| Outcome: | The proposed operation can encode positional information and confirms our hypothesis. |
Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding (2024.findings-emnlp)
Copied to clipboard
Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu
| Challenge: | Existing methods to enhance length extrapolation of large language models have been developed, but a systematic survey is lacking. |
| Approach: | They propose to examine the effects of positional encoding on length extrapolation. |
| Outcome: | The proposed methods improve the extrapolation of large language models, but they are still lacking a systematic survey. |
Self-Ensemble: Mitigating Confidence Distortion for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models exhibit a confidence distortion problem on multichoice question-answering . Self-Ensemble solves this problem by splitting the choices into several groups . |
| Approach: | They propose a method that splits LLM choices into several groups and ensembles them to reach a final decision. |
| Outcome: | The proposed method outperforms standard inference and baseline methods on MCQA. |
PEPE: Long-context Extension for Large Language Models via Periodic Extrapolation Positional Encodings (2025.findings-emnlp)
Copied to clipboard
Jikun Hu, Dongsheng Guo, Yuli Liu, Qingyao Ai, Lixuan Wang, Xuebing Sun, Qilei Zhang, Quan Zhou, Cheng Luo
| Challenge: | Long-context extension attempts to extend contextual window in pre-trained LLMs . primary method involves expanding initial positional encodings, disrupting positional learning . |
| Approach: | They propose a new extension strategy based on Rotary Position Embedding to extend contextual window in pre-trained large language models. |
| Outcome: | The proposed method can extend the contextual window in pre-trained large language models . expansion disrupts positional encodings learned during pre-training, authors show . |
A Morphology-Based Investigation of Positional Encodings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Contemporary deep learning models handle languages with diverse morphology . morphological complexity of languages is closely linked with positional encodings . |
| Approach: | They propose to use positional encodings to integrate morphological complexity into deep learning models. |
| Outcome: | The proposed model improves on 22 languages and 5 downstream tasks. |
Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that Transformers can recognize hierarchical structures without a specific positional encoding. |
| Approach: | They show that Transformers can generate hierarchical languages without a positional encoding . they also suggest that explicit positional encoders might have a detrimental effect on generalization . |
| Outcome: | The proposed model can generate hierarchical languages with respect to model size without encoding . |